Neural Computation with Winner-Take-All as the Only Nonlinear Operation

Everybody “knows” that neural networks need more than a single layer of nonlinear units to compute interesting functions. We show that this is false if one employs winner-take-all as nonlinear unit: Any boolean function can be computed by a single -winner-takeall unit applied to weighted sums of the input variables. Any continuous function can be approximated arbitrarily well by a single soft winner-take-all unit applied to weighted sums of the input variables. Only positive weights are needed in these (linear) weighted sums. This may be of interest from the point of view of neurophysiology, since only 15% of the synapses in the cortex are inhibitory. In addition it is widely believed that there are special microcircuits in the cortex that compute winner-take-all. Our results support the view that winner-take-all is a very useful basic computational unit in Neural VLSI: it is wellknown that winner-take-all of input variables can be computed very efficiently with transistors (and a total wire length and area that is linear in ) in analog VLSI [Lazzaro et al., 1989] we show that winner-take-all is not just useful for special purpose computations, but may serve as the only nonlinear unit for neural circuits with universal computational power we show that any multi-layer perceptron needs quadratically in many gates to compute winner-take-all for input variables, hence winner-take-all provides a substantially more powerful computational unit than a perceptron (at about the same cost of implementation in analog VLSI). Complete proofs and further details to these results can be found in [Maass, 2000].